● Correlates of Clinical Performance

Problem Statement and Background. One of the greatest challenges continuing to face medical educators is the development of an admissions protocol that provides valid information pertaining to the noncognitive qualities candidates possess. An innovative protocol, the Multiple Mini-Interview, has recently been shown to be feasible, acceptable, and reliable. This article presents a first assessment of the technique’s validity. Method. Forty five candidates to the Undergraduate MD program at McMaster University participated in an MMI in Spring 2002 and enrolled in the program the following autumn. Performance on this tool and on the traditional protocol was compared to performance on preclerkship evaluation exercises. Results. The MMI was the best predictor of objective structured clinical examination performance and grade point average was the most consistent predictor of performance on multiple-choice question examinations of medical knowledge. Conclusions. While further validity testing is required, the MMI appears better able to predict preclerkship performance relative to traditional tools designed to assess the noncognitive qualities of applicants. Despite widespread recognition that noncognitive (i.e., personal) qualities are important contributors to the ability of students to become competent physicians, evidence that selection tools can accurately identify candidates in possession of the desired characteristics has remained elusive. To gain admission to almost any professional school in North America, programs require candidates to complete a personal interview during which an attempt is made to assess each candidate’s interpersonal skills, motivation, and problem exploration. There has been tremendous variability in the reliability observed within studies of the interview process, some of which can be accounted for by the finding that more structured interviews (i.e., interviews that utilize standardized questions) tend to elicit more reliable judgments. However, the vast majority of these studies have been interrater reliability studies, their results indicating simply that different individuals can sometimes agree on the strength of a performance within a single interview. To gain useful information about the candidate, however, some reassurance is required that the admissions tool provides information that is generalizable beyond that particular interview. Several studies investigating the conduct of multiple encounters have demonstrated a poor correlation of candidate performance across interviews. With this as background, it should come as no surprise that interview scores have repeatedly failed tests of validity. Twenty-five years ago, Mann was able to show that interview ratings depend more on the interviewer than on the candidate. More recently, it has been shown that traditional interviews do not predict performance on objective structured clinical examinations (OSCE) either within medical school or during licensure, despite the fact that OSCEs tend to be designed, in part, to specifically address some of the noncognitive skills that interviews are expected to assess (e.g., communication skills, problem exploration etc.,). Adopting a lesson from the evaluation community—the need to collect multiple observations whenever context specificity is a concern—Eva et al. have developed a Multiple Mini-Interview (MMI) process in an attempt to dilute the impact of individual examiners and allow for a more generalizable aggregate performance rating to be assigned to candidates. Modeled after the OSCE, the MMI consists of a series of short interviews with multiple examiners. Each station is assigned a scenario pertaining to ethical issues, communication skills, collaborative abilities, or some other noncognitive quality. Greater detail regarding the methodology can be found elsewhere, but it should be noted that a series of studies have shown the MMI to be feasible, acceptable to both candidates and examiners, and reliable (G .65–.81). This article presents the first attempt to assess the validity of this new admissions protocol by examining the relationship between preclerkship performance, the MMI, and the traditional admissions protocol used by the Undergraduate MD program at McMaster University.